Image pickup control device, image pickup device, control method for image pickup device, non-transitory computer-readable storage medium

ABSTRACT

An image pickup control device, comprising: a first obtainment unit configured to obtain a picked-up image picked up by an image pickup unit; a display control unit configured to display the picked-up image on a display; a detection unit configured to detect a viewpoint region which is a region viewed by a user in the display; a second obtainment unit configured to obtain a feature amount relating to the picked-up image; and a control unit configured to switch between a first mode, in which a focus of the image pickup unit is controlled such that a subject displayed on the viewpoint region is focused, and a second mode, in which control is executed such that the focus is not changed, on a basis of the viewpoint region and the feature amount.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image pickup control device andparticularly relates to focus control.

Description of the Related Art

In recent years, trends to automation/intelligent features have advancedfor a camera. Japanese Patent Application Publication No 2004-8323discloses an image pickup device which detects a line-of-sight of a userlooking into a finder without the user's input of a position of asubject and executes focus control (focusing) on a subject intended bythe user on the basis of the detected line-of-sight.

Moreover, Japanese Patent Application Publication No. 2017-34569discloses technology of setting to a continuous AF (autofocus) mode inwhich a subject is continuously focused when a trajectory of a user'sviewpoint (viewed position) matches a trajectory of the subject in adisplayed moving image.

However, in the aforementioned technology, if an obstacle overlaps withthe subject intended by the user during photographing of a moving image,the obstacle is focused, and an unnecessary focus change occurs, whichdeteriorates a quality of the moving image. On the other hand, duringphotographing of a still image, when the subject intended by the userappears again after overlapping by an obstacle, it takes a long time forthe focus control on the subject, and a photographing chance may belost.

Furthermore, in order to cope with loss of the subject or entering ofthe obstacle, it is conceivable to set time appropriately until the AFis executed since the detection of the line-of-sight (AF response), butthe AF response cannot be changed dynamically. Therefore, bothmaintaining of AF follow-up to the same subject intended by the user andAF follow-up characteristics at switching of the subject which is an AFtarget cannot be realized at the same time.

That is, it has not been possible to continuously focus on the subjectintended by the user by a line-of-sight input.

SUMMARY OF THE INVENTION

Thus, an object of the present invention is to provide technology ofcontinuously focusing on a subject intended by a user by a line-of-sightinput.

A first aspect of the present invention is: an image pickup controldevice, comprising: at least one memory and at least one processor whichfunction as: a first obtainment unit configured to obtain a picked-upimage picked up by an image pickup unit; a display control unitconfigured to display the picked-up image on a display; a detection unitconfigured to detect a viewpoint region which is a region viewed by auser in the display; a second obtainment unit configured to obtain afeature amount relating to the picked-up image; and a control unitconfigured to switch between a first mode, in which a focus of the imagepickup unit is controlled such that a subject displayed on the viewpointregion is focused, and a second mode, in which control is executed suchthat the focus is not changed, on a basis of the viewpoint region andthe feature amount.

A second aspect of the present invention is: a control method for animage pickup device having an image pickup unit that obtains a picked-upimage and a display that displays the picked-up image, the methodcomprising: detecting a viewpoint region which is a region viewed by auser on the display; obtaining a feature amount relating to thepicked-up image; and controlling to switch between a first mode, inwhich a focus of the image pickup unit is controlled such that a subjectdisplayed in the viewpoint region is focused, and a second mode, inwhich control is executed such that the focus is not changed, on a basisof the viewpoint region and the feature amount.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of a digital camera according to anembodiment 1.

FIG. 2 is a sectional view of the digital camera according to theembodiment 1.

FIG. 3 is a schematic view of an optical system for executingline-of-sight detection according to the embodiment 1.

FIG. 4 is a view for explaining a line-of-sight detection methodaccording to the embodiment 1.

FIG. 5 is a flowchart of line-of-sight detection processing according tothe embodiment 1.

FIG. 6 is a flowchart of focus control processing according to theembodiment 1.

FIGS. 7A to 7D are views for explaining focus control according toconventional.

FIGS. 7E to 7H are views for explaining the focus control according tothe embodiment 1.

FIG. 8 is a flowchart of the focus control processing according to anembodiment 2.

FIG. 9A is a view illustrating a distance according to the embodiment 2.

FIG. 9B is a view illustrating a motion vector of a line-of-sight regionaccording to the embodiment 2.

FIGS. 10A to 10D are views for explaining the focus control according tothe embodiment 2.

FIG. 11A is a view illustrating the distance according to the embodiment2.

FIG. 11B is a view illustrating a motion vector of the line-of-sightregion according to the embodiment 2.

FIGS. 12A to 12E are views for explaining the focus control according tothe embodiment 2.

FIG. 13A is a view illustrating the distance according to the embodiment2.

FIG. 13B is a view illustrating the motion vector of the line-of-sightregion according to the embodiment 2.

FIG. 14 is a configuration diagram of a digital camera according to anembodiment 3.

FIG. 15 is a configuration diagram of a digital camera according to anembodiment 4.

FIG. 16 is a flowchart of the focus control processing according to theembodiment 4.

FIGS. 17A to 17D are views for explaining the focus control according tothe embodiment 4.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present invention will be described byreferring to the attached drawings.

Embodiment 1

(Configuration of Digital Camera): Configuration of a digital camera 100which is an image pickup device (electronic device) according to anembodiment 1 will be described by referring to FIG. 1. FIG. 1 is aconfiguration diagram illustrating major function units of the digitalcamera 100. The digital camera 100 has an imaging optical unit 101, animage pickup element 102, an A/D conversion unit 103, an imageprocessing unit 104, and a data transfer unit 105. The digital camera100 has a memory control unit 106, a DRAM 107, a non-volatile memorycontrol unit 108, a ROM 109, a recording media control unit 110, arecording media 111, a display control unit 112, a display unit 113, aCPU 114, and an operation unit 115. The digital camera 100 has aline-of-sight detection unit 120, an image pickup element 121 foreyeball, a focus detection unit 122, an illumination light source 123,an illumination light-source drive unit 124, and a line-of-sight vectordetection unit 125.

The imaging optical unit 101 forms an optical image on the image pickupelement 102. The imaging optical unit 101 has a plurality of lens groupsincluding a focus lens and an anti-vibration lens and a diaphragm.Moreover, the imaging optical unit 101 has a focus control unit 118 thatexecutes focus control and a diaphragm control unit 119 that executesexposure adjustment, camera-shake correction and the like.

The image pickup element 102 picks up an image of a subject by executingphotoelectric conversion for converting an optical image to an electricsignal (analog image signal). The image pickup element 102 includes aCCD, a CMOS sensor and the like. Moreover, the image pickup element 102includes a plurality of independent photodiodes in an exclusive pixel oreach pixel for executing image-plane phase difference AF (phasedifference AF executed on an image-plane (image pickup plane, sensorplane)).

The A/D conversion unit 103 converts the analog image signal obtainedfrom the image pickup element 102 to a digital image signal (image data;image). The image after the conversion (image data) is output to theimage processing unit 104.

The image processing unit 104 executes processing such as correction ofchromatic aberration of magnification, development processing,noise-reduction processing, geometric deformation, and resizing such asscaling to the image (image data; digital image signal). Moreover, theimage processing unit 104 has a buffer memory. Furthermore, the imageprocessing unit 104 has an image pickup correction unit that executespixel correction, black-level correction, shading correction, flawcorrection and the like to the image converted by the A/D conversionunit 103.

The data transfer unit 105 has a plurality of DMAC (direct memory accesscontroller) and executes data transfer of an image processed by theimage processing unit 104 and the like.

The memory control unit 106 causes the DRAM 107 to read/write the databy being controlled by the CPU 114 or the data transfer unit 105.

The DRAM 107 is a memory (storage medium) storing the data. The DRAM 107stores data such as a predetermined number of still images, movingimages for a predetermined time, sound and the like, a constant foroperating the CPU 114, a program and the like. Thus, the DRAM 107includes a sufficient storage capacity for storing such data.

The non-volatile memory control unit 108 reads/writes data from/to theROM 109 by being controlled by the CPU 114.

The ROM 109 is an electrically erasable/recordable memory (non-volatilememory) and can be an EEPROM or the like. The ROM 109 stores constants,programs and the like used by the CPU 114.

The recording media control unit 110 reads records of images andrecorded data with respect to the recording media 111. The recordingmedia 111 is recording media such as an SD card recording the data.

The display control unit 112 controls display of the display unit 113.The display unit 113 is a liquid crystal display or an electronicviewfinder. The display unit 113 displays the image obtained from theimage processing unit 104, a menu screen and the like. Moreover, thedisplay unit 113 obtains an image of a real-time subject (live-viewimage: picked-up image) from the A/D conversion unit 103 and displays itby control of the display control unit 112 before the photographing ofstill images and during photographing of moving images.

The CPU 114 is a control unit such as a microcomputer that controls theentire digital camera 100. The CPU 114 controls each function unit.Moreover, the CPU 114 performs calculations required at control. The CPU114 controls the image processing unit 104, the data transfer unit 105,the memory control unit 106, the non-volatile memory control unit 108,the recording media control unit 110, the display control unit 112, theoperation unit 115, the image pickup element 102 and the like through abus 116. The CPU 114 realizes each control by executing the programrecorded in the ROM 109, for example. Moreover, the CPU 114 executescontrol of a lens and a diaphragm of the imaging optical unit 101 andobtainment of information such as a focal distance.

The operation unit 115 includes operation members such as a switch, abutton, a touch panel and the like operated by a user. For example, theoperation unit 115 is used for operations of ON/OFF of a power supplyand ON/OFF of a shutter.

The bus 116 is a system bus for transmitting a control signal of eachblock mainly from the CPU 114 and the like. A bus 117 is a data bus fortransferring mainly images.

The line-of-sight detection unit 120 detects a line-of-sight directionof the user on the basis of an image of an eyeball (eye image) inputfrom the image pickup element 121 for eyeball. Details of aline-of-sight detection operation will be described later. Moreover, theline-of-sight detection unit 120 obtains a viewpoint region which is aregion where the user is viewing in the display unit 113.

The image pickup element 121 for eyeball obtains the image of theeyeball (eye image) by forming an optical image of the eyeball of theuser looking into the viewfinder. The image pickup element 121 foreyeball outputs the eye image to the line-of-sight detection unit 120.

The focus detection unit 122 calculates a lens driving amount forcontrolling the focus. The region to be focused is determined by theline-of-sight detection unit 120 and the image processing unit 104. Thefocus detection unit 122 drives/controls the focus lens with respect tothe focus control unit 118. For the calculation of the lens drivingamount, an image-plane phase difference method based on the imageobtained by the image pickup element 102 (image for focus detection),for example, can be used.

The illumination light source 123 is a light source that emits infraredlight to the user for line-of-sight detection. The infrared lightemitted from the illumination light source 123 is emitted to the eyeballof the user, and a reflection light (reflected image) in the eyeball isformed in the image pickup element 121 for eyeball. The illuminationlight-source drive unit 124 is a drive unit that controls theillumination light source 123.

The line-of-sight vector detection unit 125 calculates time-seriesmovement of the viewed position and detects it as the line-of-sightvector from the data of the line-of-sight direction of the user detectedby the line-of-sight detection unit 120.

FIG. 2 is a sectional view (explanatory view) of a cut-off housing ofthe digital camera 100 according to the embodiment 1. In FIG. 2, thesame portions as those in FIG. 1 are given the same numbers. As shown inFIG. 2, the digital camera 100 has members and components (hardware)other than each function unit as shown in FIG. 1.

A photographing lens 100A is a photographing lens in an interchangeablelens type camera. In FIG. 2, the photographing lens 100A is illustratedto have two lenses, that is, a focus lens 205 and a lens 206, therein,but it has more lenses.

A housing unit 100B is a housing unit of a main body of the digitalcamera 100. The housing unit 100B has the image pickup element 102,light sources 123 a and 123 b, a light receiving lens 201, and an ocularlens 203 therein.

The image pickup element 102 is arranged on an image forming surface ofthe photographing lens 100A. The ocular lens 203 is a lens for a user toobserve a subject image displayed on the display unit 113.

The light sources 123 a and 123 b are light sources used for detectingthe line-of-sight direction from a relationship between a reflectionimage by corneal reflection of the light source and a pupil andilluminates an eyeball 204 of the user. The light sources 123 a and 123b have infrared-emitting diodes and are arranged around the ocular lens203. The illuminated eyeball image and the image by the cornealreflection of the light sources 123 a and 123 b are transmitted throughthe ocular lens 203 and are reflected in a light divider 202. Thereflected image is formed by the light receiving lens 201 on the imagepickup element 121 for eyeball in which photoelectric element rows suchas a CCD and the like are arranged two-dimensionally.

The light receiving lens 201 positions the pupil of the eyeball 204 ofthe user and the image pickup element 121 for eyeball in acommon-benefit image forming relationship. The line-of-sight detectionunit 120 detects the line-of-sight direction by using a predeterminedalgorithm which will be described later from the positional relationshipbetween the eyeball whose image is formed on the image pickup element121 for eyeball and the image by the corneal reflection of the lightsources 123 a and 123 b.

A diaphragm 207 is a diaphragm provided on the photographing lens 100A.The diaphragm 207 is controlled by the diaphragm control unit 119. Alens drive member 210 has a drive gear and the like. A lens drive motor211 is a motor for moving the focus lens 205. A photocoupler 209 detectsrotation of a pulse plate 208 interlocking with the lens drive member210 and outputs information of the detected rotation to the focuscontrol unit 118.

The focus control unit 118 drives the lens drive motor 211 on the basisof the information of rotation of the pulse plate 208 and theinformation of the lens drive amount and moves the focus lens 205 to afocusing position. A mount contact 212 is a well-known interface betweenthe camera and the lens.

Moreover, in FIG. 2, the operation members such as a touch-panelcompatible liquid crystal, a button-type cross key and the like arearranged as the operation unit 115.

(Line-of-Sight Detection Method): Hereinafter, a line-of-sight detectionmethod will be described by referring to FIG. 3, FIG. 4, and FIG. 5.FIG. 3 is a view for explaining a principle of the line-of-sightdetection method and a schematic view of an optical system forperforming the line-of-sight detection.

In FIG. 3, the light sources 123 a and 123 b are light sources such aslight emitting diodes and emit infrared light to the user. The lightsources 123 a and 123 b are arranged substantially symmetrically to anoptical axis of the light receiving lens 201 and illuminate the eyeball204 of the user. A part of illumination light reflected on the eyeball204 forms an image by the light receiving lens 201 on the image pickupelement 121 for eyeball. In FIG. 3, positions of the light sources 123 aand 123 b, the light receiving lens 201, and the image pickup element121 for eyeball are adjusted so that the principle of the line-of-sightdetection method can be understood easily.

(A) of FIG. 4 is a schematic diagram of an eyeball image (eyeball imageprojected to the image pickup element 121 for eyeball) picked up by theimage pickup element 121 for eyeball. (B) of FIG. 4 is a diagramillustrating output intensity of the image pickup element 121 foreyeball (CCD, for example).

FIG. 5 is a flowchart illustrating line-of-sight detection processing.The flowchart in FIG. 5 is realized by the CPU 114 executing the programstored in the ROM 109 and controlling each function unit at each Step.

At Step S501, the light sources 123 a and 123 b are driven by theillumination light-source drive unit 124 and emit the infrared lighttoward the eyeball 204 of the user. The eyeball image of the userilluminated by the infrared light is formed on the image pickup element121 for eyeball through the light receiving lens 201, and photoelectricconversion is performed by the image pickup element 121 for eyeball. Bymeans of the photoelectric conversion, the eyeball image can be handledas an eye image (image signal; electric signal).

At Step S502, the image pickup element 121 for eyeball outputs anobtained eye image to the line-of-sight detection unit 120.

At Step S503, the line-of-sight detection unit 120 calculatescoordinates of corneal reflection images Pd and Pe of the light sources123 a and 123 b and a point corresponding to a pupil center c from theeye image.

Here, as illustrated in FIG. 3, the infrared light emitted from thelight sources 123 a and 123 b illuminates a cornea 301 of the eyeball204 of the user. At this time, the corneal reflection images Pd and Peformed by a part of the infrared light reflected on the surface of thecornea 301 are converged by the light receiving lens 201, form an imageon the image pickup element 121 for eyeball, and become cornealreflection images Pd′ and Pe′ in the eye image. The light from endportions a and b of the pupil 302 similarly form an image on the imagepickup element 121 for eyeball and become pupil end images a′ and b′ inthe eye image.

(A) of FIG. 4 illustrates an example of the reflection image (eye image)obtained from the image pickup element 121 for eyeball. (B) of FIG. 4illustrates brightness information (brightness distribution) obtainedfrom the image pickup element 121 for eyeball in an area a of the eyeimage illustrated in (A) of FIG. 4. In (B) of FIG. 4, assuming that ahorizontal direction of the eye image is an X-axis and a verticaldirection as a Y-axis, the brightness distribution in the X-axisdirection is illustrated. in the present embodiment, coordinates of thecorneal reflection images Pd′ and Pe′ in the X-axis direction(horizontal direction) are assumed to be Xd and Xe. Moreover,coordinates of the pupil end images a′ and b′ in the X-axis directionare assumed to be Xa and Xb.

As illustrated in (B) of FIG. 4, the brightness at an extremely highlevel can be obtained at the coordinates Xd and Xe of the cornealreflection images Pd′ and Pe′. In the area of the pupil 302 (an areabetween the coordinates Xa to Xb), the brightness at an extremely lowlevel is obtained except the positions at the coordinates Xd and Xe. Onthe other hand, in an area of luster 401 on an outer side of the pupil302 (an area of an iris image on an outer side of the pupil imageobtained by formation of light from an iris 143), a value in the middleof the aforementioned two brightness levels is obtained.

As described above, by paying attention to the brightness level, the Xcoordinates Xd and Xe of the corneal reflection images Pd′ and Pe′ andthe X coordinates Xa and Xb of the pupil end images a′ and b′ can beobtained from the brightness distribution as illustrated in (B) of FIG.4.

Moreover, if a rotation angle θx (see FIG. 3) of the optical axis of theeyeball 204 with respect to the optical axis of the light receiving lens201 is small, a coordinate Xc of the pupil center image c′ (center ofthe pupil image) obtained by formation of the light from the pupilcenter c on the image pickup element 121 for eyeball can be expressed asXc≈(Xa+Xb)/2. That is, the coordinate Xc which is an X coordinate of thepupil center image c′ can be estimated from the X coordinates Xa and Xbof the pupil end images a′ and b′. As described above, the coordinate Xcof the pupil center image c′ and the coordinates of the cornealreflection images Pd′ and Pe′ can be estimated.

At Step S504, the line-of-sight detection unit 120 calculates an imageforming magnification β of the eyeball image. The image formingmagnification β is a magnification determined by the position of theeyeball 204 with respect to the light receiving lens 201, and can beacquired by using a function of Xd-Xe which is an interval between thecorneal reflection images Pd′ and Pe′.

At Step S505, the line-of-sight detection unit 120 calculates a rotationangle of the optical axis of the eyeball 204 with respect to the opticalaxis of the light receiving lens 201. Here, the X coordinate of a middlepoint between the corneal reflection image Pd and Pe substantiallymatches the X coordinate of a curvature center O of the cornea 301.Thus, assuming that a standard distance from the curvature center O ofthe cornea 301 to the center c of the pupil 302 is Oc, a rotation angleθx of the eyeball 204 in a Z-X plane (plane perpendicular to the Y-axis)can be calculated from formula 1. It is to be noted that a rotationangle θy of the eyeball 204 in a Z-Y plane (plane perpendicular to theX-axis) can be also calculated by a method similar to the calculationmethod of the rotation angle θx.

β×Oc×sin θx≈{(Xd+Xe)/2}−Xc  formula 1

At Step S506, the line-of-sight detection unit 120 reads out correctioncoefficient data (coefficients m, Ax, Bx, Ay, By) stored in advance inthe memory 107. The coefficient m is a constant determined byconfiguration of a finder optical system of the digital camera 100 andis a conversion coefficient for converting the rotation angles θx and θyto the coordinates corresponding to the pupil center c in a visual fieldimage (image for visual recognition) in the finder. Moreover, thecoefficients Ax, Bx, Ay, By are line-of-sight correction coefficientsfor correcting individual differences in the line-of-sights, obtained byperforming a calibration work, and are stored in the memory 107 beforestarting the line-of-sight detection processing.

At Step S507, the line-of-sight detection unit 120 acquires the user'sviewpoint (position of a gazed point; viewed position) in the image forvisual recognition displayed on the display unit 113 by using therotation angles θx, θy of the user's eyeball 204. Assuming that thecoordinates (Hx, Hy) of the viewpoint are coordinates corresponding tothe pupil center c, the coordinates (Hx, Hy) of the viewpoint can becalculated from the following formula 2 and formula 3.

Hx=m×(Ax×θx+Bx)  formula 2

Hy=m×(Ay×θy+By)  formula 3

At Step S508, the line-of-sight detection unit 120 stores thecoordinates (Hx, Hy) of the viewpoint in the memory 107. Moreover, theline-of-sight detection unit 120 measures time during which the positionof the line-of-sight remains in a certain area and stores the measuredtime as gazing time in the memory 107.

The method of obtaining the coordinates of the viewpoint on the displayelement using the corneal reflection images of the light sources 123 aand 123 b has been illustrated, but this is not limiting. Thecoordinates of the viewpoint (eyeball rotation angle) may be obtained bya well-known arbitrary method from the picked-up eyeball image.

(Focus Control Processing): Hereinafter, focus control processing of thedigital camera 100 (control method of the digital camera 100) will bedescribed by using FIG. 6. FIG. 6 is a flowchart of the focus controlprocessing of the digital camera 100. The flowchart in FIG. 6 isrealized by the CPU 114 executing the program stored in the ROM 109 andcontrolling each function unit at each Step. When an operation ofinstructing the AF control by the user is performed, the focus controlprocessing is started.

At Step S601, the image pickup element 121 for eyeball obtains the eyeimage (image data) of the user to which the illumination of the lightsource 123 is emitted and outputs the eye image to the line-of-sightdetection unit 120.

At Step S602, the line-of-sight detection unit 120 detects the user'sline-of-sight (viewpoint) by the line-of-sight detection processingexplained by using the flowchart in FIG. 5. The line-of-sight detectionunit 120 calculates a viewed position (gazing point) in the live-viewimage displayed on the display unit 113 in the finder and outputs it tothe line-of-sight vector detection unit 125. Moreover, the line-of-sightdetection unit 120 obtains a viewpoint region 701 (a region viewed bythe user in the display unit 113) which is a rectangular region with apredetermined size around the user's viewed position and outputs theviewpoint region 701 to the image processing unit 104. The viewpointregion 701 does not have to be rectangular but may be a shape such as anelliptic shape, a circular shape, a polygonal shape and the like.

At Step S603, the line-of-sight vector detection unit 125 calculates(obtains) a motion vector of the user's viewpoint region 701 (movementvector of the viewpoint) from a difference between the viewed positionof a previous frame and the viewed position of a current frame andoutputs it to the CPU 114. This difference does not have to be adifference between the continuous two frames but may be a difference ina predetermined period of time such as among at least three frames.

At Step S604, the image pickup element 102 obtains a picked-up imagewhich picked up the subject (photographed region image; live-view image)and outputs it to the image processing unit 104 through the A/Dconversion unit 103. That is, at this Step, it can be considered thatthe image processing unit 104 obtains the picked-up image from the imagepickup element 102.

At Step S605, the image processing unit 104 obtains the motion vectorfrom the previous frame of the image displayed on the viewpoint region701 in the current frame (viewpoint region image), as the motion vectorof the viewpoint region image. That is, the image processing unit 104obtains the motion vector of the image from the previous frame to thecurrent frame within a range of the viewpoint region 701 in the currentframe, as the motion vector of the viewpoint region image. Here, theimage processing unit 104 calculates the motion vector of the image inthe viewpoint region 701 (viewpoint region image) as a feature amount inthe picked-up image for determining whether or not an obstacle overlapswith the subject intended by the user. The comparison between theprevious frame and the current frame (corresponding position search) inorder to obtain the motion vector of the viewpoint region image isperformed by a template matching method or the like, for example.Moreover, the image processing unit 104 outputs the motion vector of theviewpoint region image to the CPU 114.

At Step S606, the CPU 114 determines whether or not the motion vector ofthe viewpoint region 701 matches the motion vector of the viewpointregion image. That is, at Step S606, it is determined whether or not thesubject displayed in the viewpoint region 701 has changed during oneframe. The respective motion vectors are calculated on a coordinatereference of the picked-up image (live-view image) displayed by thedisplay unit 113 in the finder. And if both the difference in the sizeof the two motion vectors and the difference in the directions arewithin a different range set in advance, the two motion vectors aredetermined to match each other. When the motion vector of the viewpointregion 701 is determined to match the motion vector of the viewpointregion image, the process proceeds to Step S607, while if it isdetermined not to match, the process proceeds to Step S608.

At Step S607, the CPU 114 controls the focus detection unit 122 andexecutes the focus control (AF; movement of the focus) by drive controlof the focus lens of the focus control unit 118 so that the subjectdisplayed on the viewpoint region 701 is focused as a first mode.

At Step S608, the CPU 114 does not execute the focus control to thesubject displayed in the viewpoint region 701, does not change the focus(focus) from the previous frame but fixes it as a second mode.

At Step S609, the CPU 114 determines whether or not the user hasperformed an operation to end the image pickup (photographing). Forexample, if an operation to turn OFF the power of the digital camera 100is performed or when the operation to instruct the AF is cancelled bythe user, the focus control processing is finished. Otherwise, the imagepickup is continued, and the process returns to Step S601.

In the following, a difference in focusing (focus position) depending onpresence/absence of application of the focus control processingaccording to the present embodiment will be described by using FIGS. 7Ato 7H. FIGS. 7A to 7H illustrate frames of the live-view images(picked-up images) displayed on the display unit 113.

FIGS. 7A to 7D illustrate frames of the live-view images when the AFcontrol is executed on the basis of the user's line-of-sight all thetime. FIGS. 7E to 7H illustrate the frames of the live-view images whenthe focus control processing of the present embodiment is applied.

Moreover, FIGS. 7A and 7E illustrate a frame f1 of the live-view image,and FIGS. 7B and 7F illustrate a frame f2 which is a frame subsequent tothe frame f1. FIGS. 7C and 7G illustrate a frame f3 which is a framesubsequent to the frame f2, and FIGS. 7D and 7H illustrate a frame f4which is a frame subsequent to the frame f3.

Here, an example in which the focus control processing according to thepresent embodiment is not applied, but the focus control is executed tothe viewpoint region 701 all the time will be described. In this case,if an obstacle 703 overlaps with a front surface of a major subject 702to be intentionally focused by the user, the obstacle 703 is temporarilyfocused (focus). That is, if the obstacle 703 overlaps with the majorsubject 702 from a state where the major subject 702 is focused asillustrated in FIG. 7A, the state changes to a state where the obstacle703 is focused as illustrated in FIGS. 7B and 7C.

Thus, during photographing of moving images, an image of a frame pickedup at an unnecessary focus position is recorded, which deteriorates aquality of the moving images to be recorded. Moreover, duringphotographing of a still image, if the major subject 702 re-appears frombehind the obstacle 703, as illustrated in FIG. 7D, time required forthe focus control is prolonged, and an increase in a release time lag isconcerned about.

On the other hand, an example in which the focus control processingaccording to the present embodiment is applied to the live-view imagewill be described. If the obstacle 703 overlaps in front of the majorsubject 702 as in the frame f2 illustrated in FIG. 7F from the statewhere the focus control is executed to the major subject 702 as in theframe f1 illustrated in FIG. 7E, the motion vector cannot be obtainedfor the major subject 702 in the viewpoint region image. Thus, themotion vector of the user's viewpoint region 701 does not match themotion vector of the viewpoint region image. Therefore, in FIG. 7F, thefocus control is not executed to the viewpoint region 704, but thefocusing (focus) is fixed to the position where the major subject 702was (is) present. Similarly, in the frame f3 illustrated in FIG. 7G,too, the focusing is fixed. After that, as illustrated in FIG. 7H, ifthe major subject 702 re-appears, the focus control to the major subject702 is resumed.

As described above, in the present embodiment, when the motion vectorsof the viewpoint region and the viewpoint region image match each other,the digital camera determines that the subject displayed on theviewpoint region is the subject intended by the user, operates in thefirst mode, and focuses on the subject. On the other hand, if the twomotion vectors do not match each other, the digital camera determinesthat the subject displayed on the viewpoint region has changed from thesubject intended by the user to another subject, operates in the secondmode, and does not change the focusing (focus). That is, in the presentembodiment, it can be considered that the digital camera switchesbetween the first mode and the second mode on the basis of the motionvector of the viewpoint region and the motion vector of the viewpointregion image (feature amount based on the picked-up image).

Therefore, in the present embodiment, even if another subject overlapsin front of the subject intended by the user, focusing on anothersubject can be prevented. That is, there is no unnecessary focus change,and a problem of the deterioration in the quality of the moving imagesor the increase in the release time lag can be solved or a possibilityof occurrence of such problems can be reduced. That is, according to thepresent embodiment, the unnecessary focus control to the subject notintended by the user can be reduced, and the continuous AF to thesubject intended by the user can be performed. Thus, moving images withgood appearance can be recorded, and a loss of a photographingopportunity can be reduced by improving a response in the still-imagephotographing.

Moreover, even if the subject followed by the line-of-sight of the useris switched, if the motion vector of the viewpoint region after theswitching matches the motion vector of the viewpoint image region, thefocus control is executed to the subject after the switching. Therefore,the AF following characteristics can be also improved.

The detection method on the premise that the user looks into the finderwas described as the line-of-sight detection method, but this is notlimiting. For example, a line-of-sight when the user is looking at thedisplay on a rear-surface panel may be detected. Moreover, theprocessing at each Step of the aforementioned flowchart may be executedby dedicated hardware instead of the aforementioned function unit suchas the CPU 114.

Embodiment 2

The digital camera 100 according to an embodiment 2 will be described.In the embodiment 2, the digital camera 100 executes the focus controlprocessing by using distance information between the digital camera 100(image pickup element 102) and the subject as a feature amount of thepicked-up image, instead of the motion vector of the viewpoint regionimage.

Configuration of the digital camera 100 according to the presentembodiment is identical to the configuration of the digital camera 100according to the embodiment 1. Moreover, since a part of the focuscontrol processing of the digital camera 100 according to the presentembodiment is identical to the focus control processing according to theembodiment 1, only different portions will be described, and thedescription on the identical portion will be omitted.

FIG. 8 is a flowchart of the focus control processing of the digitalcamera 100 according to the present embodiment. The flowchart in FIG. 8is realized by the CPU 114 executing the program stored in the ROM 109and controlling each function unit at each Step. When the operation ofinstructing the AF control by the user is performed, and the focuscontrol processing is started, the processing from Step S601 to StepS604 as described by using FIG. 6 is executed.

At Step S801, the image processing unit 104 generates an image for phasedifference AF (image-plane phase difference image) from the picked-upimage (image data) obtained by the image pickup element 102. Forexample, the image processing unit 104 may extract only the data ofdedicated pixels for the phase difference AF so as to generate the imagefor phase difference AF or may generate the image for phase differenceAF configured only by each of the data of the photodiodes divided ineach pixel.

At Step S802, the image processing unit 104 obtains distance informationin the viewpoint region (information on a distance D between the digitalcamera 100 and the subject) on the basis of the image for phasedifference AF as the feature amount. Here, in the present embodiment,the distance D between the digital camera 100 and the subject is anoptical distance between the image pickup element 102 and the subject.For calculation of the distance information, if there are right andleft, that is, two in total of divided pixels in each pixel of the imagefor phase difference AF, for example, the image processing unit 104performs a correlation value calculation of a value of the left dividedpixel and a value of the right divided pixel included in the same linein the horizontal direction. Subsequently, the image processing unit 104calculates the distance D between the digital camera 100 and the subjectin an actual space on the basis of parallax between the divided pixelswith the highest correlation value, a pixel pitch of the image pickupelement 102 and the like.

At Step S803, the CPU 114 manages the information of the motion vectorof the viewpoint region and the information of the distance D in theviewpoint region as time-series data and obtains a change amount fromthe previous frame to the current frame of the time-series data (themotion vector and the distance D of the viewpoint region).

At Step S804, the CPU 114 determines whether or not a change amount ofthe motion vector of the viewpoint region is not more than a thresholdvalue TH1 set in advance and the change amount of the distance D in theviewpoint region is not more than a threshold value TH2 set in advance.If it is determined that both the two change amounts are not more thanthe threshold values, the process proceeds to Step S805, while if eveneither one of them is determined to be larger than the threshold valueset in advance, the process proceeds to Step S806.

At Step S805, the CPU 114 determines whether or not the focus control tothe viewpoint region has been executed in the previous frame (whether ornot the processing at Step S607 has been executed in the previousframe). If the focus control to the viewpoint region was executed in theprevious frame, it can be determined to be a state where the majorsubject is continuously being followed and thus, the process proceeds toStep S607. If the focus control to the viewpoint region was not executedin the previous frame, it can be determined to be a state where theobstacle overlaps in front of the major subject continues and thus, theprocess proceeds to Step S608.

At Step S806, the CPU 114 determines whether or not only the changeamount of the distance D in the viewpoint region is larger than thethreshold value. That is, it is determined whether or not the changeamount of the motion vector of the viewpoint region is not more than thethreshold value TH1. If only the change amount of the distance D in theviewpoint region is larger than the threshold value (if the changeamount of the motion vector of the viewpoint region is not more than thethreshold value TH1), the process proceeds to Step S807. Otherwise, theprocess proceeds to Step S808.

At Step S807, the CPU 114 determines whether or not the distance D inthe viewpoint region has returned to a distance Df (the distance D inthe first mode that immediately precedes) between the subject in theviewpoint region and the digital camera 100 immediately before itchanges to a value larger than a threshold value TH2. For example, if adifference between the distance D and the distance Df in the viewpointregion is within a predetermined value, it can be determined that thedistance D in the viewpoint region has returned to the value of thedistance Df. When it is determined that the distance D in the viewpointregion has returned to the value of the distance Df, the processproceeds to Step S607. Otherwise, the process proceeds to Step S608.

At Step S808, the CPU 114 determines whether or not only the changeamount of the motion vector of the viewpoint region is larger than thethreshold value. That is, it is determined whether or not the distance Din the viewpoint region is not more than the threshold value TH2. Ifonly the change amount of the motion vector of the viewpoint region islarger than the threshold value (if the change amount of the distance Dis not more than the threshold value TH2), it can be determined that theuser switched the subject to be followed or the subject with irregularmotion is being followed and thus, the process proceeds to Step S607.Otherwise, it can be determined that such a state occurs that the usertemporarily checks the surrounding situations or the informationdisplayed in the finder and thus, the process proceeds to Step S608.

The processing from Step S607 to Step S609 is similar to the processingdescribed in the embodiment 1.

As described above, in the present embodiment, in the case of thefollowing (1) to (3), the CPU 114 determines that the subject intendedto be focused by the user is not displayed in the viewpoint region anddoes not execute the focus control (operates in the second mode).Instead of the “between the continuous two frames” in the following (1)to (3), the “predetermined period among a plurality of frames and thelike” may be used.

(1) The case where the change amount of the motion vectors of theviewpoint region between the continuous two frames is not more than thethreshold value TH1 and the change amount of the distance D between thecontinuous two frames is not more than the threshold value TH2 and also,the focus control is not executed in the previous frame (operating inthe second mode). That is, the case of YES at Step S804 and NO at StepS805.

(2) The case where the change amount of the motion vectors of theviewpoint region between the continuous two frames is not more than thethreshold value TH1 and the change amount of the distance D between thecontinuous two frames is larger than the threshold value TH2, and it isdetermined that the distance D after the change has not returned to thedistance Df. That is, the case of NO at Step S804, YES at Step S806, andNO at Step S807.

(3) The case where the change amount of the motion vectors in theviewpoint region between the continuous two frames is larger than thethreshold value TH1 and the change amount of the distance D between thecontinuous two frames is larger than the threshold value TH2. That is,the case of NO at Step S804, NO at Step S806, and NO at Step S808.

On the other hand, in the case of the following (4) to (6), the CPU 114determines that the subject intended by the user is displayed in theviewpoint region and executes the focus control so that the subject inthe viewpoint region is focused (operates in the first mode). Instead ofthe “between the continuous two frames” in (4) to (6), the“predetermined period among a plurality of frames and the like” may beused.

(4) The case where the change amount of the motion vectors of theviewpoint region between the continuous two frames is not more than thethreshold value TH1 and the change amount of the distance D between thecontinuous two frames is not more than the threshold value TH2 and also,the focus control is executed in the previous frame (operating in thefirst mode). That is, the case of YES at Step S804 and YES at Step S805.

(5) The case where the change amount of the motion vectors of theviewpoint region between the continuous two frames is not more than thethreshold value TH1 and the change amount of the distance D between thecontinuous two frames is larger than the threshold value TH2, and it isdetermined that the distance D after the change has returned to thedistance Df. That is, the case of NO at Step S804, YES at Step S806, andYES at Step S807.

(6) The case where the change amount of the motion vectors of theviewpoint region between the continuous two frames is larger than thethreshold value TH1 and the change amount of the distance D between thecontinuous two frames is not more than the threshold value TH2. That is,the case of NO at Step S804, NO at Step S806, and YES at Step S808.

Here, in the present embodiment, too, similarly to the embodiment 1, thefocus control is executed in the live-view image as illustrated in FIGS.7E to 7H. The distance D in the viewpoint region and the motion vectorof the viewpoint region 701 at this time are illustrated in FIGS. 9A and9B.

At this time, even during the period when the obstacle 703 overlaps infront of the major subject 702, the user's line-of-sight continuouslyfollows the major subject 702. Thus, as illustrated in FIG. 9B, themotion vector of the user's viewpoint region 701 does not change largelyin accordance with movement of the major subject 702 but issubstantially constant.

On the other hand, as illustrated in FIG. 9A, the distance D of theviewpoint region shows a large change of the frame f2 (see FIG. 7F) inwhich the obstacle 703 overlaps in front of the major subject 702.

As described above, the motion vector of the viewpoint region 701 doesnot change largely, but if the distance D in the viewpoint regionchanges largely (YES at Step S806 and NO at Step S807), the digitalcamera 100 determines that the obstacle 703 overlaps in front of themajor subject 702. Thus, the digital camera 100 does not execute thefocus control. In the frame f3 illustrated in FIG. 7G, since there is nochange from the state where the obstacle 703 overlaps, the focusing iscontinuously fixed. In the frame f4 illustrated in FIG. 7H, a largechange occurs only in the distance D in the viewpoint region, and sinceit changes to the vicinity of the distance in the frame f1 before theprevious change (YES at Step S807), it can be determined thatoverlapping of the obstacle 703 is solved. Thus, the digital camera 100resumes the focus control to the viewpoint region 701.

(The Case of Image Pickup of the Other Live-View Images): Moreover,FIGS. 10A to 10D illustrate each of the frames of the live-view images(picked-up images) when the major subject 702 moves also in the opticalaxis direction as time elapses. FIGS. 10A to 10D illustrate the framesf1 to f4, respectively. FIGS. 11A and 11B illustrate the distance D inthe viewpoint region and the motion vectors of the viewpoint region 701.

At this time, even while the obstacle 703 overlaps in front of the majorsubject 702, the user's line-of-sight continuously follows the majorsubject 702. Thus, as illustrated in FIG. 11B, the motion vector of theuser's viewpoint region 701 does not show a large change in accordancewith the movement of the major subject 702 but is substantiallyconstant. On the other hand, the distance D in the viewpoint regionlargely changes in the frame f2 (see FIG. 10B) where the obstacle 703deeply enters as illustrated in FIG. 11A. As described above, if themotion vector of the viewpoint region 701 does not change largely andthe distance D in the viewpoint region largely changes, the digitalcamera 100 determines that the obstacle 703 overlaps the major subject702 and does not execute the focus control (fixes focusing).

Moreover, in the frame f3 illustrated in FIG. 10C, since there is nochange from the state where the obstacle 703 overlaps, the digitalcamera 100 continuously fixes focusing. In the frame f4 illustrated inFIG. 10D, a large change occurred again only in the distance D in theviewpoint region, and the distance D changed to the vicinity of apredicted value (predicted distance) on the basis of the distance Dfwhen the focus control was executed immediately before. Thus, thedigital camera 100 determines that the overlapping of the obstacle 703is solved and resumes the focus control to the viewpoint region 701. Asdescribed above, conditions for resuming the focus control are not onlythe condition that the distance D returns to the distance Df but may bea condition that the distance D changes within a predetermined valuefrom the predicted distance estimated on the basis of the distance Dwhen the focus control was executed immediately before. For example, asillustrated in FIG. 11A, the condition is considered to be satisfied ifa change rate from the distance D in the frame f1 to the distance D inthe frame f4 is within a predetermined range from the change rate fromthe distance D in the frame f0, which is one before the frame f1, to thedistance D in the frame f1.

(The Case of Image Pickup of the Other Live-View Images:) Moreover,FIGS. 12A to 12E illustrate each frame of the live-view image when theuser switches the obstacle 703 which has entered to a target of thefocus control. FIGS. 12A to 12E illustrate the frames f1 to f5,respectively. FIGS. 13A and 13B illustrate the distance D in theviewpoint region and the motion vectors of the viewpoint region 701.

As illustrated in FIG. 13B, the user's line-of-sight continuouslyfollows the major subject 702 until the frame f3 (see FIG. 12C) in whichthe obstacle 703 overlaps in front of the major subject 702. Thus, themotion vector of the user's viewpoint region 701 does not show a largechange in accordance with the movement of the major subject 702 but issubstantially constant until the frame f3. On the other hand, asillustrated in FIG. 13A, the distance D in the viewpoint region shows alarge change in the frame f3 in which the obstacle 703 overlaps with themajor subject 702. If the motion vector of the viewpoint region 701 doesnot have a large change and a large change occurs in the distance in theviewpoint region as above (YES at Step S806 and NO at Step S807), thedigital camera 100 determines that the obstacle 703 overlaps the majorsubject 702. And the digital camera 100 does not execute the focuscontrol (fixes the focusing).

Here, if the user's line-of-sight changes so as to follow the obstacle703, the viewpoint region 701 moves to a position of the subject whichmoves differently from before. Thus, in the frame f4 (see FIG. 12D), alarge change occurs in the motion vector of the viewpoint region 701without a large change in the distance D in the viewpoint region. If alarge change occurs only in the motion vector of the viewpoint region701 without a large change in the distance D in the viewpoint region asabove (YES at Step S806 and YES at Step S808), it is determined that thetarget of the focus control has been switched to the obstacle 703. Thus,the digital camera 100 resumes the focus control to the viewpoint region701.

As described above, the focus control to the viewpoint region isexecuted on the basis of the time-series change of the motion vector ofthe viewpoint region and the distance information in the viewpointregion. As a result, even if the obstacle overlaps the major subjectsimilarly to the embodiment 1, a change in the focusing point does notoccur unnecessarily. Thus, deterioration in the moving image quality oran increase in release time lag can be reduced. Moreover, even if theuser suddenly switches the target of the focus control, the focuscontrol to the viewpoint region can be resumed at once, and the subjectintended by the user can be focused.

Moreover, an example in which the focus control is executed on the basisof the time-series data of the motion vector of the viewpoint region andthe distance information in the viewpoint region has been explained, butthis is not limiting. For example, overlapping of the subject in thelive-view image may be detected by using color information or textureinformation instead of the distance information in the viewpoint region.In this case, the “change in the distance D in the viewpoint region” inthe present embodiment shall read the “change in the color or the changein the texture in the viewpoint region”, whereby the effect similar tothe present embodiment can be obtained. Moreover, by reading therecitation of the “change amount” in the present embodiment as the“change rate”, too, the effect similar to the present embodiment can beobtained. If the user's line-of-sight continuously follows the subjectaccelerating or decelerating at a constant acceleration degree, themotion vector of the viewpoint region or the change rate of the distanceD is substantially constant and thus, the reading as the “change rate”is suitable.

Embodiment 3

A digital camera 1400 according to an embodiment 3 will be described byreferring to FIG. 14. In the embodiment 3, the digital camera 1400 doesnot obtain the distance information in the viewpoint region from thepicked-up image (phase difference image) but by another distancemeasuring method.

Configuration and the focus control processing of the digital camera1400 according to the present embodiment is identical to a part of theconfiguration and the focus control processing of the digital camera 100according to the embodiment 2. Thus, in the following, only portionsdifferent from the embodiment 2 will be described, while the explanationon the identical portions will be omitted.

FIG. 14 is a configuration diagram of the digital camera 1400 accordingto the present embodiment. The digital camera 1400 has a distancemeasuring unit 1401 in addition to the function units of the digitalcamera 100 according to the embodiment 1.

The distance measuring unit 1401 obtains (measures) the distance Dbetween the subject captured in the picked-up image and the digitalcamera 1400. That is, the distance measuring unit 1401 obtains thedistance D between the subject present in an image pickup range forpicking up a picked-up image and the digital camera 1400. For example,the distance measuring unit 1401 is a distance sensor including acombination of an LED for light projection and a photodiode for lightreception required for distance measurement of Time-of-flight type or acombination of a projector and a camera required for the distancemeasurement of a pattern irradiation type.

In the present embodiment, in the focus control processing (processingin FIG. 8), instead of the processing at Step S801 and Step S802executed by the image processing unit 104, the distance measuring unit1401 obtains the distance information in the viewpoint region. The otherprocessing in the focus control processing according to the presentembodiment is the processing similar to the focus control processingaccording to the embodiment 2.

As described above, the effect similar to the embodiment 2 can beobtained by using another distance measuring method instead ofobtainment of the distance information from the image for phasedifference AF (phase difference image).

Embodiment 4

In the following, a digital camera 1500 according to an embodiment 4will be described. In the embodiment 4, the digital camera 1500 executesthe focus control to the viewpoint region on the basis of a positionalrelationship between the viewpoint region and a subject region in thepicked-up image. A part of configuration and the focus controlprocessing of the digital camera 1500 according to the presentembodiment is identical to the configuration and the focus controlprocessing of the digital camera 100 according to the embodiment 1.Thus, in the following, only portions different from the embodiment 1will be described, while the explanation on the identical portions willbe omitted.

(Configuration of the Digital Camera): FIG. 15 is a configurationdiagram of the digital camera 1500 according to the present embodiment.The digital camera 1500 has a subject detection unit 1501 inside theimage processing unit 104. On the other hand, the digital camera 1500does not have the line-of-sight vector detection unit 125 unlike theembodiment 1.

The subject detection unit 1501 detects a major subject (a specificsubject) on the basis of the picked-up image. In the present embodiment,the major subject can be a person, an animal, a ride and the likeassumed by the user to become a target of the AF. Moreover, the subjectdetection unit 1501 obtains a region of the major subject in thepicked-up image (subject region) as a feature amount in the picked-upimage. For the detection of the subject, a known art such as facedetection, human body detection, deep learning and the like can be used,for example.

(Focus Control Processing): Subsequently, the focus control processingwill be described by referring to FIG. 16. FIG. 16 is a flowchart of thefocus control processing in the present embodiment. The flowchart inFIG. 16 is realized by the CPU 114 executing the program stored in theROM 109 and controlling each function unit at each Step. When theoperation of instructing the AF control by the user is performed, andthe focus control processing is started, the processing at Step S601,Step S602, and Step S604 is executed.

At Step S1601, the subject detection unit 1501 detects a subject region1701 which is a region of the major subject in the picked-up image fromthe photographed region image (picked-up image) by using the known artas described above. The subject detection unit 1501 outputs theinformation of the subject region 1701 to the CPU 114.

At Step S1602, the CPU 114 compares the viewpoint region 701 detected atStep S602 with the subject region 1701 and determines a degree ofmatching between the two regions. The degree of matching between the tworegions is a rate of overlapping of the two regions (a rate of a size ofa region where the two regions overlap each other with respect to theentire size of the viewpoint region 701), an inverse number of thedistance between center positions of the two regions or the like.

At Step S1603, the CPU 114 determines whether or not the degree ofmatching between the viewpoint region 701 and the subject region 1701 isat least a predetermined threshold value set in advance. If the degreeof matching between the two regions is at least the predeterminedthreshold value, the process proceeds to Step S607, while if the degreeof matching between the two regions is less than the predeterminedthreshold value, the process proceeds to Step S608.

For example, FIGS. 17A to 17D illustrate a positional relationshipbetween the viewpoint region 701 and the subject region 1701. FIGS. 17Ato 17D illustrate each frame of the live-view image when the AF controlwas executed on the basis of the user's line-of-sight.

In FIG. 17A, since the detected viewpoint region 701 is included in thesubject region 1701, it is determined that the degree of matchingbetween the two regions is at least the predetermined threshold value,and the focus control is executed to the viewpoint region 701. In FIG.17B, the detected viewpoint region 701 does not overlap the subjectregion 1701 and they are independent in the positional relationship andthus, it is determined that the degree of matching between the tworegions is less than the predetermined threshold value, and the focuscontrol is not executed to the viewpoint region 701 (focusing is fixed).In FIG. 17C, since the subject region 1701 is not detected due to aninfluence of the obstacle 703, the focus control is not executed to theviewpoint region 701 (focusing is fixed). In FIG. 17D, since theviewpoint region 701 is included in the subject region 1701, it isdetermined that the degree of matching between the two regions is atleast the predetermined threshold value, and the focus control isexecuted to the viewpoint region 701.

As described above, by executing the focus control on the basis of thepositional relationship between the viewpoint region and the subjectregion, too, the effect similar to the embodiment 1 can be obtained.

The viewpoint region and the subject region may be actually displayed asrectangular frames on the display unit 113 or may be handled as internalinformation of the digital camera 1500 without being displayed.

According to the present invention, the subject intended by the user canbe continuously focused by the line-of-sight input.

Moreover, the present invention has been described in detail on thebasis of preferred embodiments thereof, but the present invention is notlimited to these specific embodiments but includes various forms in arange not departing from the gist of this invention. Furthermore, eachof the aforementioned embodiments only illustrates an embodiment of thepresent invention, and each of the embodiments can be combined asappropriate.

Moreover, the present invention can be applied not only to the imagepickup device main body but also to a control device (image-pickupcontrol device) that communicates with the image pickup device(including a network camera) through wired or wireless communication andremotely controls the image pickup device. That is, it may be an imagepickup control device that controls the image pickup element (imagepickup unit) and the display unit according to the present embodiment.The device that remotely controls the image pickup device includesdevices such as a smartphone, a tablet PC, and a desktop PC, forexample. The image pickup device can be remotely controlled by notifyinga command that causes the image pickup device to perform variousoperations and setting from the control device side on the basis of theoperation performed on the control device side or processing executed onthe control device side. Moreover, the live-view image photographed bythe image pickup device may be made displayable on the control deviceside by receiving it via the wired or wireless communication.

<Other Embodiments>: Embodiment(s) of the present invention can also berealized by a computer of a system or apparatus that reads out andexecutes computer executable instructions (e.g., one or more programs)recorded on a storage medium (which may also be referred to more fullyas a ‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2020-085890, filed on May 15, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image pickup control device, comprising: atleast one memory and at least one processor which function as: a firstobtainment unit configured to obtain a picked-up image picked up by animage pickup unit; a display control unit configured to display thepicked-up image on a display; a detection unit configured to detect aviewpoint region which is a region viewed by a user in the display; asecond obtainment unit configured to obtain a feature amount relating tothe picked-up image; and a control unit configured to switch between afirst mode, in which a focus of the image pickup unit is controlled suchthat a subject displayed on the viewpoint region is focused, and asecond mode, in which control is executed such that the focus is notchanged, on a basis of the viewpoint region and the feature amount. 2.The image pickup control device according to claim 1, wherein thecontrol unit is further configured to operate in the first mode in acase where the subject displayed on the viewpoint region is determinedto be continuously a first subject based on the viewpoint region and thefeature amount; and the control unit is further configured to operate inthe second mode in a case where the subject displayed on the viewpointregion is determined to have changed from the first subject to a secondsubject based on the viewpoint region and the feature amount.
 3. Theimage pickup control device according to claim 1, wherein the featureamount includes a motion vector of an image displayed on the viewpointregion in the picked-up image.
 4. The image pickup control deviceaccording to claim 3, wherein the control unit is further configured tooperate in the second mode in a case where a motion vector of theviewpoint region and the motion vector of the image displayed on theviewpoint region in the picked-up images are determined not to matcheach other.
 5. The image pickup control device according to claim 4,wherein the control unit is further configured to operate in the firstmode in a case where the motion vector of the viewpoint region and themotion vector of the image displayed on the viewpoint region in thepicked-up images are determined to match each other.
 6. The image pickupcontrol device according to claim 1, wherein the feature amount includesa distance between the subject displayed on the viewpoint region and theimage pickup unit.
 7. The image pickup control device according to claim6, wherein, in a case in the second mode, the control unit is furtherconfigured to continue to operate in the second mode in a case where achange amount of a motion vector of the viewpoint region in apredetermined period is not more than a first threshold value and achange amount of the distance in the predetermined period is not morethan a second threshold value.
 8. The image pickup control deviceaccording to claim 6, wherein, in a case in the first mode, the controlunit is further configured to continue to operate in the first mode in acase where a change amount of a motion vector of the viewpoint region ina predetermined period is not more than a first threshold value and achange amount of the distance in the predetermined period is not morethan a second threshold value.
 9. The image pickup control deviceaccording to claim 6, wherein the control unit is further configured tooperate in the second mode in a case 1) where a change amount of amotion vector of the viewpoint region in a predetermined period is notmore than a first threshold value, 2) a change amount of the distance inthe predetermined period is larger than a second threshold value and, 3)the distance after a change is determined not to be a value based on thedistance in the first mode that immediately precedes.
 10. The imagepickup control device according to claim 6, wherein the control unit isfurther configured to operate in the first mode in a case where 1) achange amount of a motion vector of the viewpoint region in apredetermined period is not more than a first threshold value, 2) achange amount of the distance in the predetermined period is larger thana second threshold value and, 3) the distance after a change isdetermined to be a value based on the distance in the first mode thatimmediately precedes.
 11. The image pickup control device according toclaim 9, wherein in a case where a difference between the distance afterthe change and the distance in the first mode that immediately precedesis within a first predetermined value, or in a case where a differencebetween the distance after the change and a value estimated from thedistance in the first mode that immediately precedes is within a secondpredetermined value, the control unit determines that the distance afterthe change is a value based on the distance in the first mode thatimmediately precedes.
 12. The image pickup control device according toclaim 6, wherein in a case where a change amount of a motion vector ofthe viewpoint region in a predetermined period is larger than a firstthreshold value and a change amount of the distance in the predeterminedperiod is not more than a second threshold value, the control unitoperates in the first mode.
 13. The image pickup control deviceaccording to claim 6, wherein in a case where a change amount of amotion vector of the viewpoint region in a predetermined period islarger than a first threshold value and a change amount of the distancein the predetermined period is larger than a second threshold value, thecontrol unit operates in the second mode.
 14. The image pickup controldevice according to claim 6, wherein the picked-up image includes aphase difference image; and the second obtainment unit is furtherconfigured to obtain the distance on a basis of the phase differenceimage.
 15. The image pickup control device according to claim 6, furthercomprising a distance sensor that measures the distance.
 16. The imagepickup control device according to claim 1, wherein the feature amountincludes at least one of texture information and color information on animage displayed in the viewpoint region in the picked-up image.
 17. Theimage pickup control device according to claim 1, wherein the featureamount includes a subject region which is a region on which a specificsubject is displayed in the display; and the control unit is furtherconfigured to switch between the first mode and the second mode on abasis of a degree of matching between the viewpoint region and thesubject region.
 18. The image pickup control device according to claim17, wherein the control unit is further configured to operate in thefirst mode in a case where the degree of matching is not less than apredetermined threshold value and operates in the second mode in a casewhere the degree of matching is less than the predetermined thresholdvalue.
 19. An image pickup device, comprising: the image pickup controldevice according to claim 1; and a display that displays the picked-upimage, wherein the pickup device obtains the picked-up image by pickingup an image of a subject.
 20. A control method for an image pickupdevice having an image pickup unit that obtains a picked-up image and adisplay that displays the picked-up image, the method comprising:detecting a viewpoint region which is a region viewed by a user on thedisplay; obtaining a feature amount relating to the picked-up image; andcontrolling to switch between a first mode, in which a focus of theimage pickup unit is controlled such that a subject displayed in theviewpoint region is focused, and a second mode, in which control isexecuted such that the focus is not changed, on a basis of the viewpointregion and the feature amount.
 21. A non-transitory computer-readablestorage medium storing a program for causing a computer to execute acontrol method, the control method being a control method for an imagepickup device having an image pickup unit that obtains a picked-up imageand a display that displays the picked-up image and comprising:detecting a viewpoint region which is a region viewed by a user on thedisplay; obtaining a feature amount relating to the picked-up image; andcontrolling to switch between a first mode, in which a focus of theimage pickup unit is controlled such that a subject displayed in theviewpoint region is focused, and a second mode, in which control isexecuted such that the focus is not changed, on a basis of the viewpointregion and the feature amount.